Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

Abstract Majority voting and averaging are common approaches used to resolve annotator disagreements derive single ground truth labels from multiple annotations. However, annotators may systematically disagree with one another, often reflecting their individual biases values, especially in the case of subjective tasks such as detecting affect, aggression, hate speech. Annotator capture important nuances that ignored while aggregating annotations a truth. In order address this, we investigate efficacy multi-annotator models. particular, our multi-task based approach treats predicting each annotators’ judgements separate subtasks, sharing learned representation task. We show this yields same or better performance than data prior training across seven different binary classification tasks. Our also provides way estimate uncertainty predictions, which demonstrate correlate annotation traditional methods. Being able model is useful deployment scenarios where knowing when not make prediction important.